Method for obtaining visitor information and device therefor

ABSTRACT

A method for acquiring visitor information is provided. The method includes the steps of: detecting one or more visitor objects from a video; estimating attribute information of a first visitor using feature data of a first visitor object; acquiring a perspective model of the first visitor on the basis of a travel route of the first visitor object; and acquiring interaction information of the first visitor using the perspective model of the first visitor.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of Patent CooperationTreaty (PCT) International Application No. PCT/KR2020/011194 filed onAug. 21, 2020, which claims priority to Korean Patent Application No.10-2019-0129708 filed on Oct. 18, 2019. The entire contents of PCTInternational Application No. PCT/KR2020/011194 and Korean PatentApplication No. 10-2019-0129708 are hereby incorporated by reference.

FIELD OF THE INVENTION

The present invention relates to a method and apparatus for acquiringvisit information, and more specifically to a method and apparatus foracquiring visitor information by tracking a visitor object acquired froma video.

BACKGROUND

In order to measure a customer's interest in a product in a store,information on a purchase history or a stock status of the product isused. However, in the case of an experiential store or an exhibitionstore, it is difficult to acquire accurate data for measuring interestsof customers since products are not sold. Therefore, there is a need toprovide a technique for a method of measuring a customer's interest in aproduct using a captured video of a store.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method and apparatus formeasuring a visitor's demand for a product in a store using only videodata of the store.

A visitor information acquisition method according to one embodiment ofthe invention may comprise the steps of: detecting one or more visitorobjects from a video; estimating attribute information of a firstvisitor using feature data of a first visitor object; acquiring aperspective model of the first visitor on the basis of a travel route ofthe first visitor object; and acquiring interaction information of thefirst visitor using the perspective model of the first visitor.

According to one embodiment of the invention, the step of detecting theone or more visitor objects may comprise the steps of: detecting thefirst visitor object and a second visitor object; and changing anidentification value of a second visitor to an identification value ofthe first visitor when similarity between the feature data of the firstvisitor object and feature data of the second visitor object is not lessthan a reference value.

According to one embodiment of the invention, the step of changing theidentification value of the second visitor to the identification valueof the first visitor may comprise the step of not changing theidentification value of the second visitor to the identification valueof the first visitor when the first visitor object and the secondvisitor object are detected from a single frame.

According to one embodiment of the invention, the step of detecting theone or more visitor objects may comprise the step of generating a graphcorresponding to identification value information of the one or morevisitor objects, wherein nodes of the graph correspond to the visitorobjects and edges may be generated on the basis of similarity betweenthe visitor objects corresponding to the nodes.

According to one embodiment of the invention, the step of generating thegraph may comprise the step of removing an edge between nodescorresponding to the visitor objects that cannot exist in a singleframe.

According to one embodiment of the invention, the step of generating thegraph may comprise the step of generating a plurality of groupsconsisting of one or more nodes using a graph traversal algorithm,wherein the graph traversal algorithm may preferably be a breadth-firstsearch (BFS) algorithm.

According to one embodiment of the invention, the step of generating thegraph may comprise the step of not connecting an edge between a nodecorresponding to the first visitor object and a node corresponding tothe second visitor object when the first visitor object and the secondvisitor object are detected from a single frame.

According to one embodiment of the invention, the feature data of thevisitor objects are data on feature vectors of the visitor objects, andthe similarity may be acquired by subtracting a cosine valuecorresponding to an angular distance between the feature vector of thefirst visitor object and the feature vector of the second visitor objectfrom a cosine distance 1 as below:

${distance} = {{1 - {\cos(\theta)}} = {1 - {\frac{a \cdot b}{{a} \cdot {b}}.}}}$

According to one embodiment of the invention, the perspective model is alinear model for transforming a screen coordinate system of the video toa real-world coordinate system, and the step of acquiring theinteraction information of the first visitor may comprise the steps of:acquiring a travel route of the first visitor object using theperspective model of the first visitor; converting a travel route of thefirst visitor object in the screen coordinate system into a travel routeof the first visitor in the real-world coordinate system on the basis ofthe perspective model of the first visitor; and acquiring theinteraction information of the first visitor using the travel route ofthe first visitor in the real-world coordinate system.

According to one embodiment of the invention, the step of estimating theattribute information of the first visitor may comprise the step ofacquiring pose data of the first visitor using the feature data of thefirst visitor object, and the step of acquiring the interactioninformation of the first visitor may comprise the step of usinginformation on a product corresponding to the travel route of the firstvisitor in the real-world coordinate system and the pose data of thefirst visitor to acquire information on the first visitor's interactionwith the product.

According to one embodiment of the invention, the step of acquiring theinteraction information of the first visitor may comprise the steps of:acquiring a region-specific interest index of the first visitor on thebasis of the travel route of the first visitor in the real-worldcoordinate system and a travel time; and designating a category of thefirst visitor using the region-specific interest index of the firstvisitor and the interaction information of the first visitor, whereinthe category of the first visitor includes a visiting customer, aninterested customer, an intrigued customer, and a purchasing customer.

According to one embodiment of the invention, the step of designatingthe category of the first visitor may comprise the steps of: generatinga heat map for each category of the first visitor; and generating atwo-dimensional graph from the interaction information of the firstvisitor.

A visitor information acquisition apparatus according to anotherembodiment of the invention comprises a processor that may be configuredto: detect one or more visitor objects from a video; estimate attributeinformation of a first visitor using feature data of a first visitorobject; acquire a perspective model of the first visitor on the basis ofa travel route of the first visitor object; and acquire interactioninformation of the first visitor using the perspective model of thefirst visitor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a visitor information acquisition systemcomprising a visitor information acquisition apparatus according to oneembodiment of the invention.

FIG. 2 is a hardware block diagram for illustrating a visitorinformation acquisition apparatus according to one embodiment of theinvention.

FIG. 3 is a flowchart of a visitor information acquisition methodaccording to one embodiment of the invention.

FIG. 4 specifically illustrates some of the steps shown in FIG. 3 .

FIG. 5 specifically illustrates some of the steps shown in FIG. 3 .

FIG. 6A illustrates a perspective model of a visitor according to oneembodiment of the invention.

FIG. 6B illustrates a perspective model of a visitor according to oneembodiment of the invention.

FIG. 6C illustrates a perspective model of a visitor according to oneembodiment of the invention.

FIG. 7A illustrates a travel route of a visitor according to oneembodiment of the invention.

FIG. 7B illustrates a travel route of a visitor according to oneembodiment of the invention.

DETAILED DESCRIPTION

In the following detailed description of the present invention,references are made to the accompanying drawings that show, by way ofillustration, specific embodiments in which the invention may bepracticed. These embodiments are described in sufficient detail toenable those skilled in the art to practice the invention. It is to beunderstood that the various embodiments of the invention, althoughdifferent from each other, are not necessarily mutually exclusive. Forexample, specific shapes, structures, and characteristics describedherein may be implemented as modified from one embodiment to anotherwithout departing from the spirit and scope of the invention.Furthermore, it shall be understood that the positions or arrangementsof individual elements within each embodiment may also be modifiedwithout departing from the spirit and scope of the invention. Therefore,the following detailed description is not to be taken in a limitingsense, and the scope of the invention is to be taken as encompassing thescope of the appended claims and all equivalents thereof. In thedrawings, like reference numerals refer to the same or similar elementsthroughout the several views.

Hereinafter, various embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings toenable those skilled in the art to easily implement the invention.

A visitor information acquisition system comprising a visitorinformation acquisition apparatus according to one embodiment of theinvention will be described in detail below with reference to FIG. 1 .

According to one embodiment of the invention, a visitor informationacquisition system may comprise one or more cameras 10, a videoacquisition apparatus 40, and a visitor information acquisitionapparatus 100. According to some embodiments of the invention, thevisitor information acquisition system may further comprise an NVR 30 towhich a video of the camera is transferred.

According to one embodiment of the invention, the one or more cameras 10may capture a video of a store and transfer the video to the NVR 30 viaa network 20. According to one embodiment of the invention, the videoacquisition apparatus 40 may transfer the video acquired by the camera10 to the visitor information acquisition apparatus 100 via the network20.

According to one embodiment of the invention, the visitor informationacquisition apparatus 100 may acquire video data from the videoacquisition apparatus 40. However, according to another embodiment ofthe invention, the visitor information acquisition apparatus 100 and thevideo acquisition apparatus 40 may not be implemented as physicallyseparated hardware, and according to yet another embodiment of theinvention, the visitor information acquisition apparatus 100 and thecamera 10 may also not be implemented as physically separated hardware.In this case, the video acquisition apparatus 40 or the camera 10 may beincluded in the visitor information acquisition apparatus 100 as amodule implemented as software.

According to one embodiment of the invention, the visitor informationacquisition apparatus 100 may be configured to: detect one or morevisitor objects from a video acquired from the camera 10 or the videoacquisition apparatus 40; estimate attribute information of a visitorusing feature data of a visitor object; acquire a perspective model ofthe visitor on the basis of a travel route of the visitor object; andacquire interaction information of the visitor using the acquiredperspective model of the visitor. The foregoing will be described indetail below with reference to FIGS. 3 to 7B.

According to one embodiment of the invention, the visitor informationacquisition apparatus 100 may acquire a region-specific interest indexof the visitor, so that the store may use information on a productdisplayed in each pre-specified region to measure the visitor's interestin the product and predict expected demand for the product.

Further, according to one embodiment of the invention, the visitorinformation acquisition apparatus 100 may use a travel route andinteraction information of the visitor to classify the visitor into oneof a visiting customer, an interested customer, an intrigued customer,and a purchasing customer for each region of the store. The foregoingwill be described in detail below with reference to the relateddrawings.

FIG. 2 is a hardware block diagram for illustrating a visitorinformation acquisition apparatus according to one embodiment of theinvention. The internal configuration of the visitor informationacquisition apparatus according to one embodiment of the invention willbe described in detail with reference to FIG. 2 .

According to one embodiment of the invention, the visitor informationacquisition apparatus 100 may comprise an input/output interface 140, amemory 110, a processor 120, and a communication module 130. The memory110 may be a computer-readable recording medium, and may include randomaccess memory (RAM), read only memory (ROM), and a permanent massstorage device such as a disk drive. Further, the memory 110 maytemporarily or permanently store program codes and settings forcontrolling the visitor information acquisition apparatus 100, cameravideos, and visitor information.

The processor 120 may be configured to process instructions of acomputer program by executing basic arithmetic, logic, and input/outputoperations. The instructions may be provided to the processor 120 by thememory 110 or the communication module 130. For example, the processor120 may be configured to execute received instructions according toprogram codes stored in a storage device (e.g., the memory 110).

The communication module 130 may function to communicate with anexternal server via the network. As an example, a request generated bythe processor 120 of the visitor information acquisition apparatus 100according to program codes stored in a storage device (e.g., the memory110) may be transferred to the external server via the network under thecontrol of the communication module 130. In contrast, the visitorinformation acquisition apparatus 100 may receive control signals orcommands, contents, files, or the like, which are provided under thecontrol of a processor of the external server, via the network throughthe communication module 130. For example, the control signals orcommands of the external server received through the communicationmodule 130 may be transferred to the processor 120 or the memory 110,and the contents or files may be stored in a storage medium, which maybe further provided in the visitor information acquisition apparatus100.

Further, the communication module 130 may communicate with the externalserver via the network. Although a communication scheme thereof is notparticularly limited, the network may be a local area wirelesscommunication network. For example, the network may be a Bluetooth,Bluetooth Low Energy (BLE), or WiFi communication network.

The input/output interface 140 may receive a user's input and displayoutput data. The input/output interface 140 according to one embodimentof the invention may show statistical data generated using visitorinformation on a display.

In addition, according to other embodiments of the invention, thevisitor information acquisition apparatus 100 may further include othercomponents than those illustrated in FIG. 2 . However, most ofconventional components need not be explicitly illustrated. For example,the visitor information acquisition apparatus 100 may include a batteryand a charging device for supplying power to internal components of thevisitor information acquisition apparatus. Alternatively, the visitorinformation acquisition apparatus 100 may be implemented to include atleast some of the above-described input/output devices, or may furtherinclude other components such as a transceiver, a global positioningsystem (GPS) module, various sensors, and a database.

Further, although not illustrated in FIG. 2 , one or more camera modulesmay be included in the visitor information acquisition apparatus 100.According to one embodiment of the invention, the memory 110 may receivevideo data from the camera module. According to one embodiment of theinvention, the camera module may include one or more individual cameras.For example, the camera module may be a camera module embedded in thevisitor information acquisition apparatus, or may be a module connectedwith a separately provided camera device.

FIG. 3 is a flowchart of a visitor information acquisition methodaccording to one embodiment of the invention.

According to one embodiment of the invention, the visitor informationacquisition apparatus may acquire a video (S110) and detect one or morevisitor objects from the video (S120). According to one embodiment ofthe invention, the visitor information acquisition apparatus may detecta bounding box corresponding to a visitor. According to anotherembodiment of the invention, the visitor information acquisitionapparatus may detect bounding boxes corresponding to a head and a bodyof the visitor, respectively, in order to accurately identify visitors.

Further, the visitor information acquisition apparatus according to oneembodiment of the invention may perform visitor identification with highaccuracy using a method of assigning a visitor object identificationvalue to be described below with reference to FIG. 4 .

According to one embodiment of the invention, the visitor informationacquisition apparatus may estimate attribute information of a firstvisitor using feature data of a first visitor object (S130). Accordingto one embodiment of the invention, the visitor information acquisitionapparatus may embed the first visitor object to acquire vector datareflecting the features of the first visitor object. In this case, thevisitor information acquisition apparatus may convert the first visitorobject into an element in a specific vector space to acquire featurevector data of the first visitor.

For example, the visitor information acquisition apparatus may embed ahead region and a body region of the first visitor to acquire featurevector data of the first visitor. The feature vector data of the firstvisitor acquired through the above process may be used to assignidentification values of the one or more visitor objects detected in thestep S120.

According to one embodiment of the invention, the visitor informationacquisition apparatus may estimate a perspective model of the firstvisitor on the basis of a travel route of the first visitor object(S140). According to one embodiment of the invention, the visitorinformation acquisition apparatus may estimate a perspective model ofthe first visitor for acquiring an actual travel route of the firstvisitor on the basis of the travel route of the first visitor object. Inthis case, it should be noted that the travel route of the first visitorobject is a travel route of a bounding box corresponding to the firstvisitor object based on a screen coordinate system, and the travel routeof the first visitor is a travel route of the first visitor based on areal-world coordinate system.

According to one embodiment of the invention, the visitor informationacquisition apparatus may estimate a linear perspective model whosenormal vector varies with the height of the visitor. According to theembodiment of the invention, the visitor information acquisitionapparatus may acquire a perspective model of each visitor on the basisof a line of intersection between a trend plane for each visitor objectand the xy-plane in a three-dimensional coordinate system based on thex-coordinate of a bounding box corresponding to the visitor's headregion, the y-coordinate of the bounding box corresponding to thevisitor's head region, and the z-coordinate corresponding to thevertical length of the bounding box corresponding to the visitor's headregion. The foregoing will be described in more detail below withreference to FIGS. 6A to 6C.

According to one embodiment of the invention, the visitor informationacquisition apparatus may acquire interaction information of the firstvisitor using the perspective model of the first visitor (S150).According to one embodiment of the invention, the visitor informationacquisition apparatus may acquire interaction information of the firstvisitor and a region-specific interest index of the first visitor usingthe perspective model of the first visitor.

According to one embodiment of the invention, the visitor informationacquisition apparatus may designate a category of the first visitor asone of a visiting customer, an interested customer, an intriguedcustomer, and a purchasing customer using the region-specific interestindex of the first visitor and the interaction information of the firstvisitor.

FIG. 4 illustrates a method of identifying a visitor according to oneembodiment of the invention.

According to one embodiment of the invention, the visitor informationacquisition apparatus may detect a first visitor object and a secondvisitor object (S121). According to one embodiment of the invention, thevisitor information acquisition apparatus may independently detect ahead region and a body region of a visitor in order to accurately detecta visitor object.

According to one embodiment of the invention, the visitor informationacquisition apparatus may embed the head region and body region of thevisitor to acquire feature vector data of each of the regions.

In this case, the visitor information acquisition apparatus may acquirefeature vector data for the head region of the visitor instead ofacquiring feature vector data for a face region of the visitor. When thefeature vector data for the visitor's head region is acquired accordingto the embodiment of the invention, the features of a broader regionincluding the visitor's face region may be detected. However, in thiscase, it should be noted that the feature vector data for the visitor'shead region may be data detected only from the visitor's head regionthat is frontally captured by the camera.

According to one embodiment of the invention, the visitor informationacquisition apparatus may acquire only feature vector data for thevisitor's head region that is acquired from a part of a plurality offrame images included in the video, in order to reduce the amount ofcomputing operations. The method of selecting a part of the frame imagesmay include various methods such as a method of selecting frame imagesat regular intervals, a method of selecting a randomly specified numberof frame images, or a method of selecting frame images only when thefront of the visitor object is captured.

According to one embodiment of the invention, the visitor informationacquisition apparatus may extract a head object image from the detectedhead region of the visitor, and acquire facial feature points from thehead object image. For example, the facial feature points may be one ormore of points corresponding to both eyes, a nose, and both ends oflips. According to one embodiment of the invention, the visitorinformation acquisition apparatus may acquire a head object image onlywhen all of the above facial feature points satisfy a specifiedconfidence score. The confidence score of the facial feature points maybe acquired through a computer vision algorithm that may be used bythose skilled in the art.

A method of embedding a body region of a visitor will be describedbelow. According to one embodiment of the invention, the method ofembedding the visitor's body region by the visitor informationacquisition apparatus is similar to the above-described method ofembedding the visitor's head region, while there is a problem that thebody region is frequently hidden by an external structure or anothervisitor's body unlike the head region. Accordingly, the visitorinformation acquisition apparatus according to some embodiments of theinvention may sort bounding boxes corresponding to the visitor's bodyregion in ascending order of size of areas where the bounding boxesoverlap other bounding boxes.

According to one embodiment of the invention, the visitor informationacquisition apparatus may check whether the first visitor object and thesecond visitor object have been detected from a single frame (S122).When a specific visitor moves out of the view of the camera and entersthe view of the camera again, detection of a plurality of visitorobjects for a single visitor may be performed if the plurality ofvisitor objects overlap and the visitor travels non-linearly.

Accordingly, the visitor information acquisition apparatus according toone embodiment of the invention may adjust identification valuesassigned to the visitor objects in order to accurately identify thevisitor objects. In this case, the visitor information acquisitionapparatus may adjust the identification values assigned to the visitorobjects by comparing similarity between vector data.

According to one embodiment of the invention, when the first visitorobject and the second visitor object have not been detected from asingle frame, the visitor information acquisition apparatus may changethe identification value of the second visitor object to theidentification value of the first visitor object (S123). Further,according to one embodiment of the invention, when the first visitorobject and the second visitor object have been detected from a singleframe, the identification value of the second visitor object may not bechanged to the identification value of the first visitor object becausethe first visitor object and the second visitor object are highly likelyto correspond to different person objects.

The visitor information acquisition apparatus according to someembodiments of the invention may represent visitor objects detected fromthe video in the form of a graph. In this case, the visitor informationacquisition apparatus may represent the respective visitor objectsdetected from a plurality of frame images included in the video asnodes, and represent similarity between the visitor objectscorresponding to the respective nodes as edges.

In this case, one or more visitor objects corresponding to nodes notconnected by edges may be defined as a non-coexistent combination thatcannot be simultaneously detected from a single frame. Accordingly, thevisitor information acquisition apparatus according to some embodimentsof the invention may remove edges so that nodes corresponding to asingle visitor are not connected in the graph.

FIGS. 5 illustrates a method for acquiring a travel route andinteraction information of a visitor using a perspective model of thevisitor according to one embodiment of the invention.

According to one embodiment of the invention, the visitor informationacquisition apparatus may acquire a travel route of the first visitorobject using the perspective model of the first visitor (S151). Thevisitor information acquisition apparatus may acquire a perspectivemodel of a visitor on the basis of a line of intersection between atrend plane for each visitor object and the xy-plane in athree-dimensional coordinate system with the x-coordinate of a boundingbox corresponding to the visitor's head region, the y-coordinate of thebounding box corresponding to the visitor's head region, and thez-coordinate corresponding to the vertical length of the bounding boxcorresponding to the visitor's head region.

According to one embodiment of the invention, the visitor's perspectivemodel may be acquired in the form of z=α(x+y+b), where a is a constantthat determines a normal vector of the plane and may vary with theheight of each visitor. A method of acquiring the perspective model willbe described in detail below with reference to FIGS. 6A to 6C.

According to one embodiment of the invention, the visitor informationacquisition apparatus may convert a travel route of the first visitorobject in a screen coordinate system into a travel route of the firstvisitor in a real-world coordinate system on the basis of theperspective model of the first visitor (S152). According to oneembodiment of the invention, the visitor information acquisitionapparatus may estimate height information of the visitor detected fromthe video, using the perspective model of the first visitor acquired inthe step S151. Further, the visitor information acquisition apparatusmay acquire a travel route of the visitor in the real-world coordinatesystem on the basis of the bounding box of the detected visitor objectand the estimated height information of the visitor. The foregoing willbe described in more detail below with reference to FIGS. 7A and 7B.

According to one embodiment of the invention, the visitor informationacquisition apparatus may acquire interaction information of the firstvisitor using the travel route of the first visitor in the real-worldcoordinate system and pose data of the first visitor (S153). Accordingto one embodiment of the invention, the visitor information acquisitionapparatus may match the acquired travel route of the first visitor inthe real-world coordinate system with floor plan information of a storevisited by the first visitor to acquire region information of the store.Further, according to another embodiment of the invention, the visitorinformation acquisition apparatus may further acquire information on atime for which the first visitor stays in each region. According to yetanother embodiment of the invention, the visitor information acquisitionapparatus may further acquire pose data of the first visitor to acquireinteraction information of the first visitor for products in each regionof the store.

In addition, according to one embodiment of the invention, the visitorinformation acquisition apparatus may measure a region-specific interestindex of the first visitor using the travel route of the first visitorand the interaction information of the first visitor. Thereafter, thevisitor information acquisition apparatus may designate a category ofthe first visitor on the basis of the interest index of the firstvisitor. According to one embodiment of the invention, the first visitormay be designated into one or more categories of a visiting customer, aninterested customer, an intrigued customer, and a purchasing customer.

A method of designating a category of the first visitor by the visitorinformation acquisition apparatus according to some embodiments of theinvention will be described in detail below.

The visitor information acquisition apparatus according to oneembodiment of the invention may acquire an interest index of a visitoron the basis of a travel route of the visitor. The interest index f(t)may be acquired using the following equation. Here, t is a spent timeand α is a parameter for determining a time reference value by whichinterest is determined.

$\begin{matrix}{{f(t)} = \left\{ \begin{matrix}{e^{t - \alpha},} & {0 \leq t < \alpha} \\{{\ln\left( {t - \alpha + 1} \right)},} & {\alpha \leq t}\end{matrix} \right.} & \left( {{EQ}.1} \right)\end{matrix}$

The visitor information acquisition apparatus according to oneembodiment of the invention may acquire a time for which a visitorhaving an interest index of 1 or higher stays in a specific region.According to one embodiment of the invention, when the interest index is1 or higher, the visitor detected by the visitor information acquisitionapparatus may be a visitor who stays in the specific region longer thana threshold value. However, it should be noted that the reference valueof the interest index is not limited to 1 as described above, and may bechanged as long as modification thereof is made feasible by thoseskilled in the art.

According to another embodiment of the invention, the visitorinformation acquisition apparatus may convert a visitor's path into aheat map on the basis of the visitor's travel route. According to yetanother embodiment of the invention, the visitor information acquisitionapparatus may acquire visit order information of each visitor in aspecific region. Moreover, and the visitor information acquisitionapparatus may further acquire a travel route of each visitor in thestore and resulting category information and interaction information ofthe visitor.

According to one embodiment of the invention, a visitor who isdesignated into a category of an interested customer may have aninterest index of 1 or higher in a specific region. That is, in thiscase, the visitor may have stayed in the specific region longer than athreshold value.

According to one embodiment of the invention, a visitor who isdesignated into a category of an intrigued customer may have an interestindex of 1 or higher in a specific region, and the visitor's interactionwith the specific region may be detected. In this case, the visitorinformation acquisition apparatus may further acquire information on theinteraction performed by the visitor designated into the category of theintrigued customer and information on a product with which theinteraction is performed.

According to one embodiment of the invention, a visitor who isdesignated into a category of a purchasing customer may pay for aproduct so that payment information may exist. According to theembodiment of the invention, the visitor information acquisitionapparatus according to some embodiments of the invention may usepreprocessed information corresponding to the payment information inorder to protect personal information, and may also use only productinformation excluding personal information of the visitor.

FIGS. 6A to 6C illustrate a perspective model of a visitor according toone embodiment of the invention.

A method of acquiring a perspective model of a visitor will be describedin detail below with reference to FIGS. 6A to 6C.

According to one embodiment of the invention, with respect to each ofone or more detected visitor objects, the visitor informationacquisition apparatus may acquire three-dimensional coordinates based onthe x-coordinate of a bounding box corresponding to each visitor's headregion, the y-coordinate of the bounding box corresponding to thevisitor's head region, and the z-coordinate corresponding to thevertical length of the bounding box corresponding to the visitor's headregion. In this case, the visitor information acquisition apparatus mayperform random sample consensus (RANSAC) to eliminate outliers. However,it should be noted that the method of eliminating outliers is notlimited to the RANSAC and may include various methods that may bechanged by those skilled in the art, and the outlier elimination may notbe performed in other embodiments of the invention.

Thereafter, the visitor information acquisition apparatus may performmultiple linear regression analysis on the one or more acquiredvisitors, and coefficients (i.e., weights=[w₁, w₂]) and a constant(i.e., bias=b) may be acquired from a set of the above three-dimensionalcoordinates. Further, a trend plane may be acquired through regressionanalysis, and the acquired trend plane may be expressed as EQ. 2 below.

z=w ₁ x+w ₂ y+b  (EQ. 2)

When the regression analysis is ideally performed, a trend planeaccording to some embodiments of the invention may be illustrated asshown in FIG. 6A. In this case, at least one trend plane correspondingto at least one visitor may all generate the same line of intersectionwith the xy-plane. However, since most of the actual trend planes areillustrated as shown in FIG. 6B, the visitor information acquisitionapparatus according to some embodiments of the invention may estimateone or more lines of intersection between one or more trend planes andthe xy-plane.

More specifically, one or more lines of intersection between one or moreactually generated trend planes and the xy-plane may be generated asshown in FIG. 6C. When a visitor object is detected from a 1080×1920video, the visitor information acquisition apparatus according to oneembodiment of the invention may analyze only the case in which xϵ[0,1080]. In this case, the visitor information acquisition apparatus maygenerate one or more data sets by substituting all x satisfying 0≤x≤1080into one or more lines of intersection acquired from one or more trendplanes corresponding to one or more visitor objects.

Thereafter, the visitor information acquisition apparatus may performorthogonal distance regression (ODR) to acquire a straight linerepresenting the one or more lines of intersection. By performing theODR instead of the linear regression analysis, it is possible to acquirea straight line that may geometrically represent the one or more linesof intersection. Through the foregoing, it is possible to acquire astraight line wx+y+b=0 on the xy-plane that represents the visitor'sperspective model. That is, the perspective model corresponding to eachvisitor may be expressed as z=α(x+y+b), where a may be a constant thatdetermines a normal vector of the plane corresponding to the height ofthe visitor.

FIGS. 7A and 7B illustrate a travel route of a visitor according to oneembodiment of the invention.

According to one embodiment of the invention, the visitor informationacquisition apparatus may estimate the height of the visitor using theperspective model. The visitor information acquisition apparatusaccording to the embodiment of the invention may estimate the height ofthe visitor object by a method to be described below even when someregions of a plurality of visitor objects are covered by structures orother visitor objects.

For example, it is assumed below that the visitor informationacquisition apparatus according to one embodiment of the inventionacquires a frame image in which a bounding box of a head region and abounding box of a body region are simultaneously detected for allvisitors. In this case, the visitor information acquisition apparatusmay select a body bounding box of a visitor object in which nooverlapping region exists between visitors.

Then, the visitor information acquisition apparatus may acquire avertical distance between a center point of a lower end of the bodybounding box corresponding to each of the plurality of visitors and acenter point of the head bounding box of the visitor. When thecoordinates of the body bounding box are [100, 300, 180, 580] and thecoordinates of the head bounding box are [130, 320, 160, 360], thecenter point of the lower end of the body bounding box may be located at(140, 580) and the center point of the head bounding box may be locatedat (145, 340). In this case, the visitor information acquisitionapparatus may acquire the vertical distance between the center point ofthe lower end of the body bounding box and the center point of the headbounding box as 240.

Thereafter, the visitor information acquisition apparatus may convertthe acquired vertical distance between the center point of the lower endof the body bounding box and the center point of the head bounding boxinto a vertical distance at a specified point using the perspectivemodel. Further, the height of each visitor may be estimated bycalculating an average of the vertical distance at the specified pointconverted from the vertical distance between the center points for eachvisitor.

A method of acquiring a travel route of a visitor in a real-worldcoordinate system using the visitor's perspective model and heightinformation acquired as above will be described in detail below.

According to one embodiment of the invention, the visitor informationacquisition apparatus may estimate a travel route of a visitor object ina screen coordinate system using a perspective model and heightinformation of a visitor. Thereafter, a top-view transformation may beperformed to convert the travel route of the visitor object in thescreen coordinate system into a travel route of the visitor in a floorplan serving as a real-world coordinate system. The method of top-viewtransformation is not limited to the foregoing and may be performedusing various methods that may be easily used by those skilled in theart.

More specifically, the visitor information acquisition apparatus mayfirst acquire a travel route of the visitor's head region object in thescreen coordinate system.

Further, the visitor information acquisition apparatus may acquire adistance from a center coordinate point a of the head region to a lowerend of the visitor's body region using the perspective model. As aresult, on the basis of the acquired coordinates of the lower end of thevisitor object, the travel route of the visitor object in the screencoordinate system may be acquired. Thereafter, the visitor informationacquisition apparatus may perform a top-view transformation on thetravel route of the visitor object in the screen coordinate system, andacquire the travel route of the visitor in the floor plan as shown inFIG. 7B through a perspective transformation as shown in FIG. 7A.

The above-described apparatus may be implemented with hardwarecomponents, software components, and/or a combination of the hardwarecomponents and software components. For example, the apparatus andcomponents described in connection with the embodiments of the inventionmay be implemented with one or more general-purpose computers orspecial-purpose computers, such as a processor, a controller, anarithmetic logic unit (ALU), a digital signal processor, amicrocomputer, a field programmable gate array (FPGA), a programmablelogic unit (PLU), a microprocessor, or any other device capable ofexecuting and responding to instructions. The processing device may runan operating system (OS) and one or more software applications executedon the operating system. Further, in response to the execution of thesoftware, the processing device may access, store, operate, process, andcreate data. For the convenience of understanding, it is described insome cases that one processing device is used. However, those skilled inthe art would understand that the processing device may include aplurality of pieces and/or types of processing elements. For example,the processing device may include a plurality of processors, or includea single processor and a single controller. Further, the processingdevice may have a different processing configuration such as a parallelprocessor.

The software may include computer programs, codes, instructions, or acombination of the foregoing, and may configure the processing device tooperate as desired or instruct the processing device in an independentor collective manner. The software and/or data may be permanently ortemporarily embodied in some type of machine, component, physicalequipment, virtual equipment, computer storage medium or device, ortransmitted signal wave, so as to be interpreted by the processingdevice or provide instructions or data to the processing device. Thesoftware may be distributed on computer systems connected to a network,and stored or executed in a distributed manner. The software and datamay be stored on one or more computer-readable recording media.

The method according to the embodiments of the invention may beimplemented in the form of program instructions that can be executed byvarious computer components, and stored on a computer-readable recordingmedium. The computer-readable recording medium may include programinstructions, data files, and data structures, separately or incombination. The program instructions stored on the computer-readablerecording medium may be specially designed and configured for theembodiments of the invention, or may also be known and available tothose skilled in the computer software field. Examples of thecomputer-readable recording medium include the following: magnetic mediasuch as hard disks, floppy disks, and magnetic tapes; optical media suchas compact disk-read only memory (CD-ROM) and digital versatile disks(DVDs); magneto-optical media such as floptical disks; and hardwaredevices such as read-only memory (ROM), random access memory (RAM) andflash memory, which are specially configured to store and executeprogram instructions. Examples of the program instructions include notonly machine language codes created by a compiler, but also high-levellanguage codes that can be executed by a computer using an interpreter.The above hardware devices may be configured to operate as one or moresoftware modules to perform the operations of the embodiments of theinvention, and vice versa.

Although the embodiments of the invention have been described above interms of the limited embodiments and drawings, various modifications andchanges may be made from the above description by those skilled in theart. For example, appropriate results may be achieved even if thedescribed techniques are performed in a different order from thedescribed method, and/or the components of the described system,structure, apparatus, circuit, and the like are coupled or combined, orreplaced with other components or equivalents, in a different form fromthe described method.

Therefore, other implementations, other embodiments, and equivalents ofthe appended claims will also fall within the scope of the claims.

What is claimed is:
 1. A visitor information acquisition methodperformed by a computing apparatus, comprising the steps of: detectingone or more visitor objects from a video; estimating attributeinformation of a first visitor using feature data of a first visitorobject; acquiring a perspective model of the first visitor on the basisof a travel route of the first visitor object; and acquiring interactioninformation of the first visitor using the perspective model of thefirst visitor.
 2. The visitor information acquisition method of claim 1,wherein the step of detecting the one or more visitor objects comprisesthe steps of: detecting the first visitor object and a second visitorobject; and changing an identification value of a second visitor to anidentification value of the first visitor when similarity between thefeature data of the first visitor object and feature data of the secondvisitor object is not less than a reference value.
 3. The visitorinformation acquisition method of claim 2, wherein the step of changingthe identification value of the second visitor to the identificationvalue of the first visitor comprises the step of: not changing theidentification value of the second visitor to the identification valueof the first visitor when the first visitor object and the secondvisitor object are detected from a single frame.
 4. The visitorinformation acquisition method of claim 1, wherein the perspective modelis a linear model for transforming a screen coordinate system of thevideo to a real-world coordinate system, and wherein the step ofacquiring the interaction information of the first visitor comprises thesteps of: acquiring a travel route of the first visitor object using theperspective model of the first visitor; converting a travel route of thefirst visitor object in the screen coordinate system into a travel routeof the first visitor in the real-world coordinate system on the basis ofthe perspective model of the first visitor; and acquiring theinteraction information of the first visitor using the travel route ofthe first visitor in the real-world coordinate system.
 5. The visitorinformation acquisition method of claim 4, wherein the step ofestimating the attribute information of the first visitor comprises thestep of: acquiring pose data of the first visitor using the feature dataof the first visitor object, and wherein the step of acquiring theinteraction information of the first visitor comprises the step of:using information on a product corresponding to the travel route of thefirst visitor in the real-world coordinate system and the pose data ofthe first visitor to acquire information on the first visitor'sinteraction with the product.
 6. The visitor information acquisitionmethod of claim 4, wherein the step of acquiring the interactioninformation of the first visitor comprises the steps of: acquiring aregion-specific interest index of the first visitor on the basis of thetravel route of the first visitor in the real-world coordinate systemand a travel time; and designating a category of the first visitor usingthe region-specific interest index of the first visitor and theinteraction information of the first visitor, and wherein the categoryof the first visitor includes a visiting customer, an interestedcustomer, an intrigued customer, and a purchasing customer.